United States Patent m 

Johnson 



iiiiHiiiiiiiiie 



US005664065A 
[U] Patent Number: 
[45] Date of Patent: 



5,664,065 
Sep. 2, 1997 



[54] PULSE-COUPLED AUTOMATIC OBJECT 
RECOGNITION SYSTEM DEDICATORY 
CLAUSE 

[75] Inventor: John L. Johnson, Huntsville* Ala. 

[73] Assignee: The United States of America as 

represented by the Secretary of the 
Army, Washington, D.C. 

[21] Appl No.: 668,029 
[22] Filed: Jun. 17, 1996 

[51] Int CL 6 GOGT 15/00 

[52] U.S. CL 395/21; 395/20; 395/22; 

395/25 

[58] Field of Search ... 382/158. 159; 

359/558, 576, 559; 395/22, 24, 21, 27, 

20 

[56] References Cited 

U.S. PATENT DOCUMENTS 

4,961,615 10/1990 Owechkoetal. .. 359U1 

5,167,007 U/1992 Toyoda 395/25 

5,289,30* 2/1994 Phuvan » 359^140 

5,497,253 3/1996 Stoll et al » » ...... 359/29 

OTHER PUBLICATIONS 

Johnson et aL "Observation of Periodic Waves in a pulse-c- 
oupled neural network," Optics Letters, vol. 18. No. 15, pp. 
1253-1255 Aug. 1, 1993. 



Johnson, "Pulse-coupled neural nets: translation, rotation, 
scale, distortion, and intensity signal invariance for images," 
Applied Optics, vol. 33, No. 26., pp. 6239-6253 Sep. 10, 
1994. 

"Feature Linking via Synchronization among Distributed 
Assemblies: Simulations of Results from Cat Visual Cor- 
tex" by R. Eckhorn, H. J. Reitboeck. M. Arndt and P. Dicke, 
Neural Computation 2, pp. 293-307 (1990). 

Primary Examiner— Tariq R. Hafiz 

Assistant Examiner — Ji-Yong D. Chung 

Attorney. Agent, or Firm— Hugh P. Nicholson; Freddie M. 

Bush; Hay Kyung Chang 



[57] 



ABSTRACT 



The pulse-coupled automatic recognition system processes 
information utilizing parallel neural network and transmits 
information to and from the processing sites as time signals 
which encode in their pulse phase structure the geometrical 
content of the spatial distributions of light. A locating device 
then measures the coordinates of the spatial distributions 
(pulsating segment) and displays the coordinates of the 
pulsating segment on a screen for observation. The pulsating 
segment indicates the presence and location of the compo- 
nent of the input scene that corresponds to the segment 

18 Claims, 3 Drawing Sheets 



215-^ , 225 s 
239 \L 



227 



237-^ 



235 ,201 



115 



213 



217 219 



205 




209 



04/18/2004, EAST Version: 1.4.1 




04/18/2004, EAST Version: 1.4.1 



U.S. Patent Sep. 2, 1997 sheet 2 of 3 5,664,065 




04/18/2004, EAST Version: 1.4.1 



U.S. Patent Sep. 2, 1997 Sheet 3 of 3 



5,664,065 




04/18/2004, EAST Version: 1.4.1 



5,664,065 

1 2 

PULSE-COUPLED AUTOMATIC OBJECT and/or temporal groups of pulses from other neurons and 

RECOGNITION SYSTEM DEDICATORY algebraically multiplying the result at multiplier HI by the 

CLAUSE feeding input F y that comes from a standard video camera. 

The resulting product U, is then used as the modified input 

The invention described herein may be manufactured, 5 to the pulse generator The modulating linking input which 

used and licensed by or for the Government for goveramen- travels by channel 109 is preprocessed prior to the biasing 

tal purposes without the payment to me of any royalties and multiplication that happen at summing junction 113 and 

thereon. multiplier 111, respectively. 

(3) Receptive Field Structures: 

BACKGROUND OF THE INVENTION 10 The aforementioned pre-processing is done by receptive 

Jt . field structures 115 of the model neuron. The temporal 

Prior art in computers consists of analog and digital preprocessing usually consists of convolving the input 

processors. The analog devices implement algebra and cal- pulses Y (K1) through Y(Kn) from other neurons (first to nth 

cuius functions for the purpose of solving systems of ©qua- ncur ons) with exponential decay time kernels 119 through 

tions while digital devices implement arithmetical and logi- l5 nx m ^.0^ relaxation processes and are 

cal statements in a serial fashion and do not use parallel identical to that used in threshold element 107 except that 

processing. mc characteristic decay time of time kernels 119 through 

The pulse-coupled neural network (PCNN), derived from 121 is shorter man that of threshold element 107. Sub se- 
ttle Eckhorn mathematical linking field model (see "Feature q UeD t spatial pre-processing consists of forming a weighted 
Linking via Synchronization among Distributed Assemblies: 20 sum, at summing junction 113 (which also adds the linking 
Simulations of Results from Cat Visual Cortex,** by R. strength (5), of pulses over a given spatial region. This 
Eckhorn. H. J. Reitboeck. M. Arndt and R Dicke, Neural weighted sum is the linking input fi^L,. 
Computation 2. pp. 293-307 (1990)), is a higher order As illustrated in FIG. 1, the receptive field structures 
neural network which uses pulses and pulse coupling to consist of means for receiving the pulses from other model 
form linking waves and time signals. It presents a general 25 neurons and a source of video image and combining them 
method of combining information, transmitting information into appropriate input signals for the linking mechanism, 
to and from processing sites as time signals. The time signals Each model neuron can have many receptive fields, both 
encode in their pulse phase structure the geometrical content feeding and linking* and the several receptive fields can each 
of (he input spatial distributions and other feature informa- receive a pulse from the same transmitting neuron at the 
tion. It combines, or fuses, the information from separate 30 same time. The receptive field structures receive many 
sources by multiplying their time signals to form a compos- inputs and generates only one output, Uj. A receptive field 
ite time signal The PCNN allows time-synchronous pulse can have a single or multiple connection points each of 
activity as well as parallel processing through neural net- which is available for connection with one input pulse 
work algorithms. It contains both sums and products that can signal. A multiplicative weight (not illustrated) can be 
be combined to make logical rules for information fusion. 35 applied to the amplitude of the input pulse and the value of 

The pulse-coupled neural network model neuron has three the weight can be different for every connection point The 
parts: (1) a pulse generator (2) a linking mechanism and (3) connection point can further have a first-order relaxation 
receptive field structures. The network itself, as distin- process which acts on the weighted input pulse and produces 
guished from the individual neuron, is composed of many an exponentially-weighted time average. All of the first- 
such neurons all interconnected with each other and further 40 order relaxation processes can have different and distinct 
connected to external signal sources and external systems. decay time constants and all of the receptive field connection 
With reference to FIG. 1 which is a diagram of a model points can have different time decay constants. Following 
neuron, each part of the neuron is explained below in detail the connection point weighting and the first-order decay 
In FIG. 1. as in the other figures, like numbers represent like processes, the signals are linearly summed at summing 
parts and arrows denote the direction of signal travel. ' 45 junction 113 and biased to become modulating linking input 

(1) The Pulse Generator: l+ftly. 

As shown, I^c generator 101 consists of threshold SUMMARY OF THE INVENTION 
discriminator 103 followed by pulse former 105. The dis- 
criminator is triggered when an input signal U,, called the Hectro-optical pulse-coupled neural networks generate 
internal activity, exceeds the value of the threshold signal 6 y 50 pulse patterns from optical images of cluttered input scenes, 
at some point in time. When such an event occurs, the The pulse-coupled automatic object recognition system uti- 
aiscrirainator sends a signal to the pulse former causing the lizes the pulse patterns to detect and recognize pre-selected 
latter to produce spike-like pulse trains. These pulses are objects in such cluttered environments. Time signals that are 
produced at a fixed rate and arc all identical. The pulse produced from these pulse patterns are characteristic of the 
former output is the neuron's output, Y* which feeds toother 55 various components of the input scene. The time signals that 
neurons and which also feeds back to threshold element 107. correspond to pre-selected objects are used selectively to 
The threshold element is a leaky integrator, modeled by a open a time gate thereby transmitting tight only when the 
first-order relaxation or exponential decay process. It is pre-selected objects are present in the scene, 
recharged by the pulses produced by the pulse former until DESCRIPTION OF THE DRAWING 
it is driven above the level of the internal activity at which 60 

time the pulse generator stops making pulses. FIG. 1 is a diagram of a model neuron. 

(2) The Linking Mechanism: FIG- 2 shows a new electro-optical implementation of the 
Linking mechanism 117 allows an input signal, called the neuron based on the principles of Eckhorn as explained 

modulating linking input l-*-pyLy, to modulate the standard above. 

input, called the feeding input F,, to pulse generator 101 . The 65 FIG. 3 illustrates a preferred embodiment of the automatic 

modulating linking input is produced by adding, at summing object recognition system utilizing the electro-optical impie- 

junction 113, a bias offset to L, which is derived from spatial mentation of the neuron as depicted in FIG. 2 above. 
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DESCRIPTION OF THE PREFERRED coated glass plate 231 which itself is located at Ac 

EMBODIMENT re-imaging plane and third SLM 225 and is further posi- 

~ f - nn nrttu te% T7i/~* <> ^ rf™^,^ <>~a nft u^ tioned such that the focal point of the first summing lens 

Refemng now to FIG. 2 the structure and operation of the 6etcaninc$ mc ^ md loc ^on of such sub _ sct 

decfrc^cal^^ on SLM 225. The first summing lens may be set at the focal 

ron based on the Eckhcrn model is explained in detail. plane or deliberately set out-of-focus. If it is set out-of- 

Feeding optical input originating from a standard video focus, then each point of the image Y y on second glass plate 
camera (not shown here) is input to first spatial light 231 receives contribution from several points of the image 
modulator (SLM) 201 and when linking optical input $JLj on the other neurons 215 of third SLM 225. A multiplier 
(from a source to be detailed later) passes through SLM 201, 10 mask 227 that may be positioned anywhere between third 
it multiplies with F^ thereby producing U,. U, is then SLM 225 and second glass plate 231 multiplies the optical 
transmitted to be incident on reimaging lens 203 which, in image of Y, emerging from the third SLM with linking 
turn, focusses the image onto first detector array 205. The strength factor* p. If the mask is set arbitrarily out-of-focus. 
first detector array, which may be a CCD array, converts the then each point of the mask will multiply by its particular 
optical image of U, into corresponding electronic video 15 value the several points of the image on the other neurons 
signal and inputs this electronic U, to step function element 215. Thus the mask's transmissivity at each point gives 
207 which performs die functions of the discriminator in the some type of weights. However, for the special case where 
Eckhora model neuron. If the electronic U, exceeds the the mask is located at the aperture of first summing lens 229, 
value of the threshold signal 8, that is input to the step the mask wQl give the identically weighted receptive field 
function element from second detector array 209. then the ^ for all the points in the image field of the lens. From second 
step function element goes from 'XT to "1". This action glass plate 231, the resultant image which is the weighted 
causes pulse former 211 which is coupled to the step sum out of the receptive field structures (as depicted in FIG. 
function element to produce at least one pulse. However, 1) travels to beamsplitter 233 that provides a bias offset by 
usually multiple pulses are produced by the pulse former in guiding into the optical system a beam of a given constant- 
spike-like pulse trains, one pulse per frame time. These 25 intensity from third suitable light source 237. The biased 
pulses, thusly generated by the pulse former, are the pulse image, now referred to as 1+f^U. is men re-imaged by 
pattern output, \ p of the jth neuron. Y y is then simulta- second summing lens 235 onto first SLM 201 where it 
neously fed into second SLM 213 and a set 215 of other multiplies with feeding optical input F, to result in modied 
neurons surrounding or in the vicinity of the jth neuron, input Uy. Thereafter, XJj is processed as described above. 

The processing of Y, that goes to second SLM 213 is 30 In the electro-optical implementation of the pulse -coupled 

discussed first: While the pulse pattern Y y is visible on the neuron as illustrated in FIG. 2. the source of optical feeding 

second SLM, a light from first suitable source 217 is shone input F y and third light source 237 for providing the bias are 

through the SLM, thereby reimaging Y y onto phospher- always on, independently of the other elements of the neural 

coated glass plate 221 via re-imaging lens 219. Upon network. The neural network is self-contained and the pulse 

receiving a lit-up pulse, the phospher-coated glass plate 35 pattern Y, the combined output of neurons 215. that is 

brightens then subsequently undergoes a slow decay. When imaged onto third SLM 225 is the final output of the 

Y y is a pulse train, what emerges from glass plate 221 is a pulse-coupled neural network (PCNN). 

pattern of pulses, all decaying in a staggered fashion. These This final output Y of the PCNN can be used in a system 

decaying pulses are re-imaged by third re-imaging lens 223 for automatic recognition of selected objects in a cluttered 

onto second detector array 209, the video data stream into 40 environment FIG. 3 shows a diagram of the preferred 

which is perfectly synchronized with the data stream from embodiment of such a system. 

first detector array 205 to step function element 207. The The pulse pattern Y A on PCNN 200A modulates colli- 

second detector array, which may be a television camera, mated light beam 301 passing therethrough from first light 

reads the decaying pulses and sends corresponding clec- panel 323. The resulting optical image of Y A impinges on 

tronic video signals, 6^ to the step function element 45 first focussing lens 303 which focusses the image to a point 

received by the step function clement from second detector on single-pixel detector 305 which, in turn, produces elec- 

array 209. is subtracted from U, received from first detector tronic output signals in response to the focus sed input image, 

array 205. If the result is "(T or negative, then nothing These electronic output signals are the reference time signals 

occurs at the step function element However, if the sub- of the various components of pulse pattern Y^ and are 

traction yields a positive result (i.e. U^B,), then the step so coupled into time correlator 307 that is suitable for recording 

function element goes from "0" to M l at that instant and and storing the reference time signals and is further capable 

pulse former 211, that is coupled to the step function of transmitting to time gate 313 the time signals that 

element produces at least one pulse output Y^. The step correspond to pre- selected components of the entire pulse 

function clement is brought back down to 44 0 when the pattern Y A . 

pulses produced by me pulse former while the step function 55 Contemporaneously with the production of the time 
element remains in is enough to result in a new 0, that signals, another collimated beam of light 309 from second 
is greater than U^. In such a case, the subtraction of 9, from light panel 325 impinges on PCNN 200B and is modulated 
Uj would result in a negative sum thereby bringing the step by pulse pattern Y B before being incident on second focus- 
function element from "1" to "0 W . smg lens 311. It is noted here that Y„ and Y* are identical 
As stated earlier. Y y is also fed into other neurons (pixels) 60 to each other and PCNN*s200A and 200B have the identical 
215 which surround or are in the vicinity of the jth neuron. number and geometrical arrangement of pixels. Upon inci- 
all of which neurons being the pixels of third SLM 225. dence of the image of Y*. the second focussing lens, then. 
When receptive field structures 115 of a given sub-set of focusses the image onto time gate 313 which opens and 
neurons receive Yy, light from second suitable source 239 is closes in phase with the time signal input thereto from time 
incident on SLM 225 and transforms Y, into an optical 65 correlator 307, thereby selectively transmitting the focus sed 
image which is then imaged onto first surnming lens 229. light onto collimating lens 315. Such a time gate can be 
Hie first surnming lens is located between second phospher- single-pixel optical spatial light modulators. The collimating 
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leas, then, re-coUimates the light and transmits it to impinge 
on video image detector array 317, such as a CCD array, that 
has the identical Dumber and geometrical arrangement of 
pixels as do the first and second PCNN's. The output of 
detector array 317 is further input to centroid locator 319 
which measures the coordinates oft he light distribution on 
the detector array and displays the information on screen 
321. Since time gate 313 opens only in response to the 
reference time signal input from time correlator 307 and the 
input reference time signals are those of a pre-selected 
component of the pulse pattern Y (corresponding to a 
pre-selected object in the original scene that is represented 
by the pulse pattern), the output light distribution on detector 
array 317 is primarily that of the pre-selected object image. 
Therefore, the coordinates produced by centroid locator 319 
correspond to and are indicative of the location of that 
pre-selected object in the original scene and the displayed 
location of the pre-selected object bears the same relation to 
the screen as does the object itself to the entire scene. 

Although a particular embodiment and form of this inven- 
tion has been illustrated, it is apparent that various modifi- 
cations and embodiments of the invention may be made by 
those skilled in the art without departing from the scope and 
spirit oft he foregoing disclosure. Accordingly, the scope of 
the invention should be limited only by the claims appended 
hereto. 25 

I claim: 

1. A pulse-coupled automatic object recognition system 
for detecting and recognizing pre-selected objects in a scene, 
said system comprising: a first and a second pulse-coupled 
neural networks (PCNN's), said PCNN's being capable of ^ 
producing corresponding electronic pulse patterns in 
response to optical inputs thereto, said optical inputs being 
descriptive of the scene; a means for producing reference 
time signals, said time signals being characteristic of the 
portions of the scene that correspond to said time signals; a 
means for detecting light; a gating means simultaneously 
coupled between said second PCNN, said light-detecting 
means and said time signal-producing means for selectively 
allowing, in response to time signal inputs from said signal- 
producing means, the transmission of light from said second 
PCNN onto said light-detecting means, said transmitted 40 
light being indicative of the pre-selected object and a 
measuring means coupled to said light-detecting means to 
measure the coordinates of light distribution on said light- 
detecting means, thereby yielding the positional information 
of the pre-selected object in the input scene. 

2. A pulse-coupled automatic object recognition system as 
set forth in claim 1, wherein said first and second PCNN's 
have the identical number and geometrical arrangement of 
pixels. 

3. A pulse-coupled automatic object recognition system as 
set forth in claim 2, wherein said system further comprises 
a means to illuminate the pulse patterns on said first and 
second PCNN's, thereby producing optical images of said 
pulse patterns. 

4. An automatic object recognition system as set forth in 
daim 3. wherein said means for producing reference time 
signals comprises a single-pixel detector capable of gener- 
ating time signals in response to optical inputs thereto; a first 
focussing lens positioned to receive the optical image of the 
electronic pulse patterns from said first PCNN and focus the 
image onto said single-pixel detector, and a time correlator, 
said correlator being coupled to said single-pixel detector to 
receive therefrom and store the time signals, said correlator 
further transmitting pre-selected time signals. 

5. An automatic object recognition system as set forth in 
daim 4, wherein said light-detecting means is a charge- 
coupled device (CCD) array. 



6. An automatic object recognition system as set forth in 
claim 5, wherein said gating means comprises a second 
focussing lens positioned to receive from said second PCNN 
the optical image of the electronic pulse patterns; a colli- 
mating lens for coUimating incident light and transmitting 
the light to impinge on said CCD array; and a time gate, said 
gate being coupled to said time correlator and being further 
positioned between said second focussing lens and coUimat- 
ing lens selectively to open, in response to time signal inputs 
from said time correlator, thereby selectively allowing the 
passage of focussed light from said second focussing lens 
onto said collimating lens. 

7. An object recognition system as set forth in claim 6, 
wherein said measuring means is a centroid locator. 

8. An object recognition system as set forth in claim 7, 
wherein said system still further comprises a means for 
displaying, said displaying means being coupled to said 
centroid locator for receiving from said locator the posi- 
tional information of the pre-selected object and displaying 
the information for visual observation. 

9. An object recognition system as set forth in claim 8, 
wherein said first, second PCNN's and said CCD array all 
have the identical number and geometrical arrangement of 
pixels. 

10. A pulse-coupled automatic object recognition system 
for locating and recognizing pre-selected objects in a scene, 
said system comprising: 

a first and a second pulse-coupled neural networks 
(PCNN's). said PCNN's being capable of generating 
corresponding electronic pulse patterns in response to 
optical inputs thereto, said optical inputs being descrip- 
tive of the scene and said pulse patterns modulating any 
beam of light passing therethrough; a single-pixel 
detector for producing time signals in response to input 
images focussed thereonto; a first focussing lens posi- 
tioned between said first PCNN and said detector to 
receive the modulated light emanating from said first 
PCNN and focus said modulated light to be incident on 
said detector; a time correlator coupled to said single- 
pixel detector for receiving therefrom the time signals, 
said time correlator being capable of further transmit- 
ting pre-selected time signals that correspond to pre- 
selected objects in the scene; a second focussing lens 
positioned to receive and focus the modulated light 
emanating from said second PCNN; a collimating lens; 
a time gate located between said second focussing lens 
and collimating lens to receive the focussed modulated 
tight from said second focussing lens and transmit the 
focussed light to be incident on said collimating lens, 
said lime gate being further coupled to said time 
correlator to receive therefrom pre-selected time sig- 
nals and opening selectively in response to the pre- 
selected time signals; a detector array for detecting the 
coUimated light incident thereon from said collimating 
lens; and a centroid locator coupled to measure the 
coordinates of the light distribution on said detector 
array to provide the locational information of the 
pre-selected objects in the scene. 

11. A pulse-coupled automatic object recognition system 
as set forth in claim 10. wherein each of said first and second 

60 PCNN's is illuminated by its own light source. 

12. An automatic object recognition system as set forth in 
claim 11. wherein said system further comprises a screen 
connected to said centroid locator for receiving the object 
locational information from said locator and displaying the 
position of the object on said screen. 

13. An automatic object recognition system as set forth in 
claim 12. wherein said display of the pre-selected object 
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image has equivalent position with respect to the entire 
screen as does the object with respect to the entire input 
scene. 

14. An object recognition system as set forth in claim 13, 
wherein said first second PCNN*s and said detector array all 3 
have the identical number and geometrical arrangement of 
pixels. 

15. An electro-optical pulse-coupled neural network 
(PCNN) having a plurality of pixels* said PCNN compris- 
ing: a first spatial light modulator (SIM) for receiving and 10 
modulating input light passing therethrough; a first detector 
array for detecting light incident thereon and producing 
electronic video signals in response to the light; a first 
re-imaging lens positioned linearly between said first SLM 
and said first detector array for receiving the modulated light 1 5 
from said first SLM and focussing the light to be incident on 
said first detector array; a second detector array; a threshold 
discriminator coupled between said first detector array and 
said second detector array to produce a causal signal in 
response to the inputs from said first and second detector 20 
arrays; a pulse former coupled to said discriminator to 
generate at least one output pulse in response to said causal 
signal; a threshold means coupled to receive said output 
pulse and produce therefrom a threshold signal, said thresh- 
old means comprising a first phospher-coated glass plate; a 25 
second SLM coupled to receive said output pulse from said 
pulse former; a first illuminator positioned to illuminate said 
second SLM; a second re-imaging lens located to focus the 
image of said output pulse from said second SLM onto said 
first phospher-coated glass plate, said first plate exhibiting 30 
phospher decay in response to the focussed image of said 
output pulse; and a third re-imaging lens positioned to focus 
the decaying image onto said second detector array, said 



second detector array producing the threshold signal in 
response to said decaying image, said threshold means 
further transmitting said threshold signal to said discrimi- 
nator; and a means for summing the output pulses of several 
of the pixels to produce a linking input suitable for inputting 
to said first SLM. 

16. An electro-optical PCNN as described in claim 15, 
wherein said summing means comprises a third SLM 
coupled to receive said output pulse from said pulse former; 
a second illuminator suitable for illuminating said third 
SLM; a second phospher-coated glass plate capable of 
undergoing phospher decay in response to image focussed 
thereon; a first summing lens coupled between said third 
SLM and said second glass plate to perform summation of 
the output pulses of a pre- selected set of several pixels of 
said third SLM and focus the sum onto said second glass 
plate to cause the phospher decay thereon; a second sum- 
ming lens; a bias positioned between said second glass plate 
and said second summing lens to add a r^e- determined bias 
to said decaying image, said second summing lens focussing 
said biased image onto said first SLM. 

17. An electro-optical PCNN as described in claim 16, 
wherein said summing means further comprises an intensity 
mask, said mask movably positioned between said third 
SLM and said second glass plate to manipulate the size of 
said pre-selected set of several pixels. 

18. An electro-optical PCNN as described in claim 17, 
wherein said bias comprises a beamsplitter located between 
said second glass plate and said second summing lens and a 
source of constant-intensity light, said source being suitably 
positioned to illuminate said beamsplitter. 
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